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ABSTRACT 

Intelligent  Adaptive  Interface  (IAI)  is  a  design  tool  that  guides  the  development  of  an  operator  interface  in 
enhancing  human-machine  system  performance.  With  a  set  of  design  principles,  this  paper  introduces  the 
concept  of  IAI  and  its  design  framework,  which  include  methodologies  for  developing  various  systems 
including  operator  interfaces.  These  methodologies  are  demonstrated  in  a  case  study  where  IAI  framework 
was  used  in  various  design  processes:  mission  scenario  analysis,  IAI  function,  task  and  goal  analyses, 
identification  of  IAI  agents,  design  of  IAI  components  in  a  task  network  model  for  simulations,  and 
implementation  and  evaluation  of  IAI  prototypes.  The  efficacy  of  IAI  was  investigated  in  the  context  of 
controlling  multiple  Uninhabited  Aerial  Vehicles  (UAVs).  This  case  study  proved  that  IAI  framework  and  the 
associated  design  principles  are  effective  in  guiding  the  design  of  an  operator  interface.  These  principles  are 
being  applied  in  the  development  of  an  operator  interface  for  the  control  of  extremely  agile  micro-aerial 
vehicle  employed  in  urban  operations.  They  will  be  further  used  to  guide  the  design  of  multi-modal  operator 
interfaces  for  the  control  ofUAV  swarming.  Additionally,  these  applications  will  provide  empirical  evidence 
to  validate  IAI  design  principles  and  thus  enhance  the  robustness  of  this  design  tool. 


1.0  INTRODUCTION 

As  automation  technologies  become  critical  elements  of  military  activities,  a  robot  revolution  is  upon  us.  It  is 
a  revolution  in  war,  like  the  invention  of  the  atomic  bomb.  More  specifically,  uninhabited  systems  (UVs)  are 
robotic  systems  that  are  becoming  more  and  more  prevalent  in  all  theatres  of  operations  (land,  sea,  air,  and 
urban).  These  robotic  systems  do  not  affect  the  “how”  but  the  “who”  of  war  fighting  at  its  fundamental  level. 
It  changes  and  redefines  the  experience  of  the  warrior  and  even  his  or  her  very  identity.  However,  the  role  of 
the  human  remains  paramount  and  the  interaction  of  humans  and  this  type  of  technological  aid  is  critical  to 
mission  accomplishment.  Net-enabled  effects-based  operations  will  place  ever-increasing  emphasis  on  the 
design  of  agile,  intelligent,  and  socio-technical  systems  that  allow  more  effective  and  efficient  acquisitions 
and  concept  developments.  Thus,  the  interaction  with  these  robotic  systems  is  vital.  The  key  concepts  here  are 
“interaction”  and  “integration”,  human-robot  interaction  and  human  systems  integration.  If  not  enough 
attention  is  given  to  this  very  topic,  then  we  may  face  social,  ethical,  and  moral  issues  when  confronting  an 
entirely  new  species:  robots.  The  challenge  of  Human-Robot  Interaction  and  Human  Systems  Integration  is  to 
develop  innovative  tools,  techniques,  and  methods  that  seamlessly  work  with  the  systems  engineering  and 
concept  development  processes  to  truly  integrate  knowledge  of  human  capabilities  and  limitations  [48]. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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1.1  The  Issue 

Despite  the  pressing  needs  for  effective  socio-technical  systems,  there  is  still  a  lack  of  well-established  design 
guidelines  for  these  human-machine  systems,  especially  for  advanced  operator  interfaces  used  by  decision¬ 
makers.  The  operator  interface  plays  critical  roles  for  dynamically  managing  enormous  amounts  of  data  and 
information  in  a  complex  and  networked  environment  (e.g.,  network  centric  warfare).  This  correlates  to  an 
impediment  of  overall  human-machine  system  performance.  Additionally,  a  lack  of  integration  between  the 
Human  Factors  (HF)  and  Human  Computer  Interaction  (HCI)  domains  has  increased  the  tendency  for 
terminology  to  become  ambiguous  and  misleading  when  applied  globally.  Thus,  there  is  a  need  to  develop  a 
unified  framework  to  guide  both  researchers  and  practitioners  in  designing  effective  human-machine  systems 
to  enhance  overall  system  performance. 

To  address  the  issue,  Defence  Research  &  Development  Canada  has  completed  a  multi-year  research  program 
to  develop  a  generic  design  framework  to  guide  the  design  of  operator  interface  in  enhancing  overall  human- 
machine  system  performance.  The  efficacy  of  the  framework  has  been  assessed  in  the  context  of  operator 
interface  design  for  the  control  of  multiple  UVs  from  an  airborne  platform  [22,23].  The  framework  is  being 
used  to  guide  the  design  of  operator  interface  for  the  control  of  single  micro-aerial  vehicle  employed  in  urban 
operations  [18,21],  To  further  validate  the  design  principles,  the  IAI  framework  is  being  applied  and  assessed 
to  aid  multi-modal  interface  designs  for  UV  swarming.  This  paper  describes  this  framework  and  a  case  study 
with  a  summary  of  research  results. 

1.2  Intelligent  Adaptive  Interface 

Operator  Machine  Interface  (OMI)  technologies  have  been  developed  in  different  forms:  from  conventional 
interfaces  to  adaptive  interfaces,  and  then  to  intelligent  adaptive  interfaces.  In  their  most  general  form, 
conventional  operator-machine  interfaces  are  the  medium  between  operators  and  a  particular  machine,  device, 
computer  program  or  other  complex  tools.  The  OMI  facilitates  the  manipulation  of  a  system  and  machine  to 
produce  the  operator’s  desired  effects. 

The  term  “intelligent  adaptive  interface”  (IAI)  is  adopted  from  intelligent  interface  and  adaptive  interface 
concepts  [28,30,32,35,38,39,40,53,59].  Intelligent  interface  intends  to  capture  the  wide  range  of  issues  and 
methodologies  from  the  application  of  intelligence  to  user  interfaces,  whereas  the  term  adaptive  user  interface 
is  used  by  many  researchers  to  emphasize  the  adaptation  of  the  interface  [1,3,4,5,7,16,27,33,36,37,42,46,57], 
Tomlinson  et  al.  [56]  defined  adaptive  interface  as  a  manner  to  predict  the  features  a  user  would  find  desirable 
and  customizable.  Thus,  an  intelligent  interface  should  tailor  its  parameters  to  certain  prescribed  specifications 
or  convert  itself  and  adjust  to  changing  circumstances,  requirements  or  needs.  In  other  words,  it  has  adaptive 
capabilities  (to  the  user  and  environment). 

Adaptive  interfaces  are  designed  to  individualize  or  personalize  systems,  thereby  increasing  the  system’s 
flexibility  and  effectiveness  [20].  Thus,  they  are  also  known  as  personalizations  to  enhance  operator 
interaction  with  a  system  by  making  the  system  more  efficient,  effective  and  easy  to  use.  The  interface  is 
adapted  (e.g.,  menu  content)  with  the  aim  of  matching  its  content  to  changing  task-related  circumstances  (e.g., 
according  to  the  mode  selected  and  application  used).  The  system  controls  whether  the  adaptation  occurs,  how 
it  occurs,  along  with  the  amount  of  adaptation  that  occurs.  However,  the  operator  does  not  fully  control  how 
the  system  adaptation  is  initially  configured. 
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Combining  the  concepts  of  intelligent  and  adaptive  interfaces,  an  IAI  can  be  defined  as  a  user  interface  with 
intelligence  that  can  perceive,  understand,  reason,  and  infer  relevant  changes  of  environments  and  users,  and 
adapt  accordingly  to  facilitate  its  interaction  with  users.  IAIs  are  OMIs  that  dynamically  change  their  control 
and/or  display  characteristics  in  reaction  to  a  task  and/or  operator  states  in  real-time  [25].  The  important  issues 
to  be  addressed  in  IAIs  [20]  are: 

•  To  formulate  reliable  and  cost-effective  IAI  development  methods; 

•  To  develop  the  tools  that  enable  easy  development  and  maintenance  of  the  intelligent  parts  of  the 
system; 

•  To  better  understand  how  and  when  intelligence  can  substantially  improve  the  interaction  (design 
practice);  and 

•  To  establish  usability  principles  for  IAIs  that  do  not  lead  users’  expectations  astray. 

IAIs  are  epitomized  by  Microsoft’s  Office  Assistant.  This  feature  was  included  in  Microsoft  Office  1997  and 
subsequent  versions  until  Office  2007.  A  ‘popular’  Office  Assistant  [52,6]  is  “Clippit”  or  “Clippy”,  which  is  a 
default  animated  paperclip.  This  feature  is  an  entry  point  to  the  help  system,  presenting  various  help  search 
functions  and  offering  advice  based  on  Bayesian  algorithms  [31].  Clippit  will  open  when  the  program  thinks 
that  a  user  requires  assistance.  It  will  modify  the  formatting  of  the  document  and  content  of  the  menus 
accordingly.  For  example,  when  a  user  types  an  address  followed  by  “Dear”,  a  prompt  Clippit  is  opened  and 
states  “It  looks  like  you’re  writing  a  letter.  Would  you  like  help?”  The  algorithms  would  use  a  combination  of 
task-based  (e.g.,  how  a  letter  is  usually  formatted)  and  user-based  (e.g.,  how  many  mistakes  the  operator  has 
made  trying  to  write  a  letter)  models  to  modify  the  interface  to  match  the  user’s  needs  and  requirements. 
Similar  to  adaptive  interfaces,  the  amount  of  adaptation  that  occurs  is  controlled  entirely  by  the  system, 
although  the  user  has  some  control  over  how  the  system  adaptation  is  initially  configured  (e.g.,  the  user  can 
turn  off  Clippit). 

Findlater  &  McGrenere  [14]  conducted  an  empirical  study  on  adaptive  user  interfaces  by  comparing  three 
menu  conditions:  static,  adaptable  and  adaptive.  Adaptive  interfaces  dynamically  adjust  the  interface  in  a  way 
that  is  intended  to  support  the  user  (system  controlled).  By  contrast,  adaptable  interfaces  provide 
customization  mechanisms  but  rely  on  the  user  to  apply  those  mechanisms  to  initiate  the  adaptation  (operator 
controlled).  Each  menu  was  implemented  as  a  split  menu  but  differed  in  the  way  the  customization  was 
implemented.  Results  indicated  that  the  static  menu  was  significantly  faster  than  the  adaptive  menu  due  to  its 
simple  processing.  The  adaptable  menu  was  also  significantly  faster  than  the  adaptive  menu  due  to  its  ease-of- 
use.  The  majority  of  operators  preferred  the  adaptable  menu  because  it  is  user-friendly.  The  implication  to  the 
Human  Factors  (HF)  and  the  Human  Computer  Interface  (HCI)  communities  is  to  determine  which  design 
approach  (adaptive  or  adaptable  interfaces)  is  the  best.  Some  argue  that  easy-to-use  predictable  mechanisms 
should  be  provided  to  keep  users  in  control  of  the  system.  Others  believe  that  if  the  right  adaptive  algorithm  is 
found,  then  operators  will  be  able  to  focus  on  their  tasks  rather  than  managing  their  tools.  Thus,  easy-to-use 
mechanisms  are  not  sufficient  for  effective  customization  (adaptive)  and  examples  should  also  be  provided  to 
users  for  guidance  on  using  the  customization  feature.  Therefore,  the  lack  of  consistent  design  approaches  in 
HF  and  HCI  domains  presents  a  challenge  to  designers  to  develop  effective  socio-technical  systems. 
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2.0  IAI  CONCEPTUAL  FRAMEWORK 

To  develop  a  unified  design  framework  encompassing  both  HF  and  HCI  approaches,  a  thorough  literature 
review  was  conducted  to  examine  existing  methodologies  in  the  design  of  intelligent  and  adaptive 
interfaces/systems.  The  results  are  summarized  in  this  section  where  the  design  principles  are  synthesized, 
condensed  and  reorganized.  There  are  three  categories  to  make  them  easy  to  follow:  adaptation  (A), 
interaction  (I),  and  organization  (O),  or  simply:  AIO  principles.  A  unified  IAI  framework  was  developed  to 
describe  detailed  design  principles  using  consistent  and  unambiguous  terminology.  A  variety  of  analytical 
approaches  to  the  development  of  an  intelligent  and  adaptive  agent-based  system  are  also  examined.  These 
design  approaches  are  CommonKADS  (Knowledge  Acquisition  and  Design  Structuring)  [51],  IDEF 
(Integrated  Computer  Aided  Manufacturing  Definition)  standards  [44],  Explicit  Models  Design  [10], 
Ecological  Interface  Design  [58],  etc. 

2.1  Adaptation  (A) 

The  following  design  principles  are  synthesized  from  the  literature  related  to  adaptations 
[17,19,43,34,29,50,47,14], 

A1 :  Appropriate  adaptation  cycles  are  required  for  automation  to  be  turned  on/off  over  a  period  of  time. 

A2:  An  IAI  should  dynamically  change  its  level  of  autonomy  according  to  the  changes  of  the  environment 
(including  operator  and  task  domains). 

A3:  Different  types  of  reasoning  tasks  should  be  performed  to  solve  problems  in  a  variety  of  domains 
using  a  number  of  problem-solving  techniques  (e.g.,  logical  inferences). 

A4:  An  IAI  must  be  able  to  reason  a  number  of  concurrent  activities. 

A5:  An  IAI  should  be  able  to  integrate  information  over  time  to  produce  an  accurate  assessment  of  current 
situations. 

A6:  An  IAI  should  be  able  to  explain  its  behaviour  in  the  time  available. 

A7 :  An  IAI  should  have  the  capability  to  predict  the  effect  of  individual  differences  on  task  efficiency. 

A8:  An  IAI  should  allow  operators  to  override  the  system's  choice  of  interaction  mode  and  choose  a  mode 
that  he/she  prefers. 

A9:  An  IAI  should  understand  the  human’s  actions  relative  to  what  it  believes  the  operator  is  doing. 

2.2  Interaction  (I) 

The  following  design  principles  are  synthesized  from  the  literature  related  to  interactions  between  users  and 
systems  [17,43,34,15,45,50,54], 

II :  Training  in  the  use  of  IAI  is  necessary. 

12:  The  human  operators  should  be  able  to  rapidly  reconfigure  an  IAI  in  response  to  changes  in  goals  and 
problem-solving  strategies. 

13:  An  operator  should  identify  why  the  required  level  of  accuracy  could  not  be  achieved  as  this 
information  may  affect  future  decisions. 

14:  An  IAI  should  make  an  operator  feel  ‘in  control’,  even  when  the  IAI  is  performing  tasks. 
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15:  An  IAI  should  be  able  to  generate  and  recognize  plans. 

16:  An  IAI  should  monitor  interaction  for  errors  in  performance  and  suggest  a  corrective  action,  which 
should  be  based  upon  an  understanding  of  the  error  source  and  the  probable  intent  of  the  operator  in 
that  context. 

17:  Displays  and  controls  should  operate  within  a  metaphor  that  is  consistent  with  the  operator’s 
conceptual  or  ‘mental’  model. 

18:  The  interface  should  provide  as  much  information  as  possible  about  any  problem  that  could  not  be 
solved,  such  that  the  operator  can  judge  the  best  course  of  action  in  the  situation. 

2.3  Organization  (O) 

IAIs  share  responsibility,  authority  and  autonomy  over  system  behaviour  with  a  human  operator.  The  aim  of 
IAIs  is  to  reduce  operator  workload  and  increase  situation  awareness.  Although  it  is  desirable  for  human 
operators  to  remain  in  charge  (adaptable),  they  cannot  be  fully  in  charge  of  all  system  operations  in  today’s 
complex  systems.  More  specifically,  humans  cannot  control  the  complex  systems  in  the  same  way  that  they 
have  been  in  earlier  cockpits  and  workstations.  Therefore,  the  following  principles  are  related  to  the  task 
allocations  between  an  operator  and  a  machine  [41,47,55,2,12,13]. 

01:  Task  and/or  decision  allocation  should  be  completed  according  to  the  interaction  of  both  mission 
goals  and  human/machine  capabilities. 

02:  IAI  should  initiate  the  off-loading  of  the  task,  and  the  operator  should  initiate  the  re -capturing  of  the 
task. 

03:  An  IAI  should  reduce  cognitive  workload  by  sharing  the  initiative  for  human-system  dialogue. 

04:  An  IAI  should  support  high  levels  of  automation,  while  simultaneously  allowing  operators  to  remain 
in  the  decision  loop  and  have  an  interactive  role  in  the  systems. 

05:  An  IAI  should  support  the  operator  by  monitoring  input  data  for  situations  that  indicate  that  the 
operator  should  be  alerted,  which  suggests  significant  changes  in  the  situation.  The  operator  should  be 
able  to  specify  what  the  conditions  may  be. 

06:  An  IAI  system  should  provide  an  operator  with  information  that  presents  progressively  more  details 
as  the  situation  demands  and  the  operator  should  be  able  to  take  control. 

07 :  An  IAI  should  provide  operators  with  coordination  of  activities  that  are  aimed  at  performing  certain 
tasks  and  achieving  specific  goals. 

08:  An  IAI  should  provide  operators  with  well-defined  organization  and  structure,  with  members  taking 
specific  roles  with  associated  power,  authority  and  status,  whilst  exhibiting  conformity  and 
commitment  to  team  norms  and  goals. 

09:  An  IAI  should  volunteer  information  to  the  operator  and  make  responses  appropriate  to  the  operator’s 
intent. 

2.4  IAI  Conceptual  Framework 

To  conduct  IAI  designs,  a  framework  is  needed  to  indicate  high  level  requirements  of  a  system.  A  framework 
also  shows  the  major  components  and  the  relationships  among  the  components.  By  synthesizing  the  above 
mentioned  AIO  design  principles,  Hou,  et  al.  [25]  developed  a  conceptual  framework  as  illustrated  in  Figure 
1 .  This  framework  provides  a  comprehensive  and  efficient  means  of  IAI  design.  The  output  of  these  processes 
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is  the  construction  and  specification  of  a  number  of  models  that  are  used  to  construct  an  intelligent  adaptive 
system  including  an  IAI: 


•  Organization  Model  (08).  This  model  incorporates  knowledge  relating  to  the  organizational  context 
in  which  the  knowledge -based  system  is  intended  to  operate  (e.g.,  command  and  control,  structures, 
Intelligence  Surveillance,  Target  Requisition  and  Reconnaissance  -  ISTAR  etc.); 

•  Task  Model  ( 01-3).  This  model  incorporates  knowledge  relating  to  tasks  and  functions  undertaken  by 
all  agents,  including  the  operator; 

•  Agent  Model  (A6-7,  A9,  15,  04-5,  09).  This  model  incorporates  knowledge  relating  to  the  participants 
of  a  system  (i.e.,  computer  and  human  agents),  as  well  as  their  roles  and  responsibilities; 


*  OMI  Design  Guidelines 

*  Automation-design  Principles 


[  *  Knowledge  of  mission  plans/goals 
i  *  Knowledge  of  mission  time-lines 

i 

i  *  Knowledge  of  mission  tasks/activities 

i 

*  Mission  Plan/Goal  Monitoring 

*  Control  (task/activities)  Monitoring 

*  Sensor  (environment)  Monitoring 

SITUATION 

““  ASSESSMENT 

Compare 

■W  OPERATOR 
—  STATE 

ASSESSMEN 

-- . M 

*  Psychophysiological  Monitoring 

*  Behavioural  Monitoring 

i  Model  of  human  cognition 

i 

1  *  Model  of  human  control  abilities 

i 

■  *  Model  of  human  communication 


Figure  1.  IAI  Conceptual  Framework  [25] 

•  User  Model  (A7-9,  11-4,  05,  09).  This  model  incorporates  knowledge  of  human  operators’  abilities, 
needs  and  preferences; 

•  System  Model  (A4-5,  15-6).  This  model  incorporates  knowledge  of  the  system’s  abilities,  needs,  and 
the  means  by  which  it  can  assist  the  human  operator  (e.g.,  advice,  automation,  interface  adaptation); 

•  World  Model  (Al-3,  05-8).  This  model  incorporates  knowledge  of  the  external  world,  such  as 
physical  (e.g.,  principles  of  flight  controls),  psychological  (e.g.,  principles  of  human  behaviour  under 
stress),  or  cultural  (e.g.,  rules  associated  with  tactics  adopted  by  hostile  forces)  environments; 
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•  Dialogue/Communication  Model  (15-8).  This  model  incorporates  knowledge  of  the  manner  in  which 
communication  takes  place  between  the  human  operator  and  the  system,  and  between  the  system 
agents  themselves; 

•  Knowledge  Model  (A3,  A5-7,  16).  This  model  incorporates  a  detailed  record  of  the  knowledge 
required  to  perform  the  tasks  that  the  system  will  be  performing;  and 

•  Design  Model  (A8,  17-8).  This  model  comprises  the  hardware  and  software  requirements  related  to  the 
construction  of  the  intelligent  adaptive  system.  This  model  also  specifies  the  means  by  which  the 
operator’s  state  is  monitored. 

3.0  ANALYTICAL  METHODOLOGIES 

To  build  a  complex  system  like  IAI,  high-level  design  principles  must  be  combined  with  thorough  analytical 
methodologies.  These  methodologies  are  the  tools  used  to  analyze  and  build  various  models  shown  in  Figure 
1.  These  models  represent  system  components  or  functions  at  different  operational  levels.  The  analytical 
processes  and  relationships  among  various  methodologies  and  models  are  illustrated  in  Figure  2.  These 
models  should  enable  system  functions  to: 

•  Modify  OMI  to  handle  the  interaction  and  dialogue  between  an  operator  and  system  components  or 
software  agents; 

•  Track  the  operator’s  goals/plans/intents  (and  progress  towards  them); 

•  Monitor  the  operator  state; 

•  Monitor  the  world  state;  and, 

•  Provide  knowledge  of  the  effects  of  system  advice,  automation  and/or  adaptation  on  the  operator  and 
world  states  (i.e.,  closed-loop  feedback). 

Additionally,  the  models  illustrated  in  Figure  2  can  be  mapped  back  onto  the  generic  IAI  conceptual 
framework  in  Figure  1 .  These  models  are  necessary  to  effectively  facilitate  operator-system  interaction.  The 
User  Model  enables  the  monitoring  of  the  operator’s  behaviour  and  physiological  states.  The  Task,  System, 
and  World  Models  enable  the  monitoring  of  mission  plan/goal  completion,  tasks/activities,  and  entities  and 
objects  in  the  external  environment.  The  Knowledge  Model  enables  the  system  to  provide  advice  to  the 
operator,  automate  tasks,  or  adapt  to  the  OMI.  The  Dialogue  Model  enables  an  effective  interaction  between 
the  system  and  the  operator.  The  generation  of  these  models  is  one  of  the  implementation  steps  of  the  generic 
conceptual  framework.  Figure  2  also  illustrates  various  analytical  approaches  which  can  contribute  to  the 
creation  of  different  models,  specifically: 

•  Cognitive  Analysis  Methodologies,  contribute  to  the  construction  of  the  Task,  Agent  and  User 
Models; 

•  Task  Analysis  Methodologies,  contribute  to  the  construction  of  the  Task,  Agent,  System  and  World 
Models; 

•  Human-Machine  Function  Allocation  and  Agent-based  Design  Principles,  contribute  to  the 
construction  of  the  Agent,  Dialogue  and  Communication  Models; 

•  Human-Machine  Interaction  and  Organization  Principles,  contribute  to  the  construction  of  the 
Dialogue  and  Communication  Models; 
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•  Human  Factors  and  Human  Computer  Interaction  Principles,  contribute  to  the  construction  of  the 
OMI  and  related  systems.  The  design  process  might  also  include  principles  from  Ecological  Interface 
Design. 

•  IDEF5  Guidelines,  contribute  to  the  construction  of  the  ontology  and  knowledge  base.  This  is  then 
used  to  enumerate  the  knowledge  captured  by  the  analysis  process; 

•  Domain  Feasibility,  Cost-Benefit  Analysis  and  Principles  for  Closed-Loop  Implementation,  contribute 
to  the  construction  of  the  Design  Model,  including  the  means  by  which  operator  state  is  monitored; 
and 


•  Human  Factors  and  Human  Computer  Interaction  Principles,  contribute  to  the  construction  of  the 
OMI  and  related  systems.  The  design  process  might  also  include  principles  from  Ecological  Interface 
Design. 

CommonKADS  Ecological  Interface  Design  Explicit  Models  Design 


Construction 
Organization  Model 
[ISTAR  /  C2  Structures] 

i 


Construct 
Task  Model 
[Top  Level  Functions] 
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Adapt 

Construct 

Task  Model 

Task  Model 

[EID  Functional  Decomposition] 

[Detailed  Functions  /  Tasks  /  Goals] 

T 

Cognitive  Task  Analysis 
Team  Cognitive  Task  Analysis  / 
Applied  Cognitive  Task  Analysis 
Cognitive  Work  Analysis 
Applied  Cognitive  Work  Analysis 

Hierarchical  Task  Analysis 
Mission  Function  &  Task  Analysis  <  • 
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System  Model 
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'  *1  World  Model 


Specify 

Communication  Model 
[Interaction  /  Organization] 
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t _ _ _ '' _ i _ 

-  * 
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Figure  2.  The  Analytical  Process  for  the  Development  of  Intelligent  Adaptive  Interface 
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Most  analytical  approaches  are  generic  (i.e.,  context  independent)  and  scalable.  The  selection  of  these 
analytical  tools  is  less  critical  as  they  can  be  (and  sometimes  must  be)  modified  to  suit  the  domain.  Further, 
these  analytical  approaches  can  be  combined  to  play  to  their  strengths  and  mitigate  weaknesses.  Several 
criteria  can  be  used  to  determine  which  analytical  and  design  tools,  techniques,  and  methodologies  should  be 
used  to  design  a  specific  intelligent  adaptive  interface: 

•  Project  constraints:  schedule  and  budget. 

•  Domain:  complexity,  criticality,  uncertainty,  and  environmental  constraints  (particularly  relevant  to 
the  choice  of  operator  state  monitoring  systems). 

•  Operator:  consequences  of  error  and  overload,  what  kind  and  quantity  of  support  is  needed,  who 
needs  to  be  in  control  (particularly  relevant  in  combat  domains). 

•  Tasks:  suitability  for  adaptation,  assistance  or  automation. 

4.0  CASE  STUDY:  IAI  FOR  THE  CONTROL  OF  MULTIPLE  UAVS 

4.1.  The  Requirement  of  Interface  Design  for  the  Control  of  Multiple  UAYs 

UAV  control  is  operator  intensive  and  can  involve  high  levels  of  workload.  As  the  quantity  and  variety  of  data 
collected  increase,  the  workload  for  UAV  operators  increases  significantly.  Moreover,  the  allocated  data  must 
be  integrated  and/or  converted  into  information  and  then  disseminated  to  those  operators  who  make  decisions. 
In  the  recent  past,  data  collection,  data  fusion,  information  management  and  distribution,  intelligence 
collecting,  and  data-related  decision-making  have  threatened  to  become  a  bottleneck.  This  situation  is  made 
even  more  complex  by  increasing  joint  operations,  and  rapid  and  flexible  warfare.  Feedback  from  the  UAV 
operators  indicate  that  improvements  in  the  operator  interface  aspect  of  these  emerging  systems  would  reap 
significant  gains  in  system  performance  and  effectiveness.  This  applies  for  both  effective  control  of  UAVs,  as 
well  as  management  of  data  and  efficient  dissemination  of  the  associated  information.  The  level  of  automation 
to  be  applied  to  the  decision-making  processes  is  a  key  aspect  facing  both  tactical  commanders  and  UAV 
system  managers.  As  a  result,  supporting  technologies  that  combine  operators  with  automation  to  satisfy 
mission  requirements  need  to  be  investigated. 

4.2  Issues 

The  control  of  multiple  UAVs  is  a  complex  cognitive  task  with  high  workload.  Applying  the  above  mentioned 
IAI  framework  into  the  design  of  these  advanced  human-machine  systems  will  likely  provide  a  real  potential 
for  improvement  in  the  effectiveness  of  UAV  control.  To  investigate  the  efficacy  of  the  IAI  framework,  the 
IAI  associated  design  principles  were  applied  into  operator  interface  design  for  the  control  of  multiple  UAVs 
from  an  airborne  platform.  The  issues  to  be  addressed  in  this  case  study  include  the  components  of  the 
framework  that  are  required  to  analyze  scenarios,  missions,  goals  and  whether  the  framework  is  effective  in 
conducting  the  development  of  the  UAV  control  system. 

4.3  The  Use  of  IAI  Framework  as  Design  Guidance 

IAIs  are  applied  here  for  reducing  operator  workload  and  improving  decision  effectiveness  in  the  employment 
and  operation  of  UAVs.  IAIs  are  human-machine  interfaces  that  aim  to  improve  the  efficiency,  effectiveness, 
and  naturalness  of  human-machine  interaction  by  acting  adaptively  and  proactively  in  response  to  external 
events  based  on  internal  mission  requirements.  In  the  context  of  UAV  control,  an  IAI  is  driven  by  software 
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agents  that  support  the  decision-making  and  action  requirements  of  operators  under  different  levels  of 
workload  and  task  complexity.  The  IAI  manifests  itself  by  presenting  the  right  information,  action  sequence 
proposals,  or  by  performing  actions  at  the  right  time.  In  addition  to  reducing  workload  for  humans  involved  in 
UAV  missions,  IAI  is  seen  as  an  opportunity  to  reduce  manning  requirement  (e.g.,  moving  from  ratio  of  10 
operators  controlling  one  UAV  to  one  operator  controlling  10  UAVs). 

4.4  How  the  Framework  Was  Used 

As  a  standard  systems  engineering  approach,  the  IAI  framework  was  used  from  the  starting  point  of  the 
operator  interface  design:  analysis.  For  systems  where  human  functions  are  predominantly  “cognitive”,  the 
method  of  analysis  should  capture  this  essential  human  activity.  IAI  framework  was  used  in  the  following 
design  processes: 

First,  operational  mission  scenario  generation.  A  fictitious  counter-terrorism  scenario  was  developed  for  a 
maritime  patrol  task  along  the  east  coast  of  Canada.  The  scenario  involved  a  UAV  crew  controlling  multiple 
UAVs  from  airborne  platform  CP140,  which  is  a  maritime  patrol  aircraft.  The  UAV  crew  consisted  of  three 
operators:  UAV  pilot  (UP),  sensor  operator  (UO),  and  tactical  navigator  (TN).  The  UAV  pilot  was  responsible 
for  the  safe  and  appropriate  conduct  of  all  UAVs  under  the  crew’s  control.  The  UO  was  responsible  for 
managing  the  information  being  returned  by  the  sensors  onboard  the  UAVs  under  the  crew’s  control,  and  for 
relating  findings  based  on  that  sensor  data  back  to  the  rest  of  the  crew  as  appropriate.  The  TN  was  ultimately 
responsible  for  overall  operations.  According  to  the  nature  of  tasks  each  crew  member  performed,  three  levels 
of  the  task  complexity  were  assigned:  low  (UP),  medium  (UO),  and  high  (TN).  Depending  on  the  number  of 
UAVs  to  control,  workload  was  also  assigned  at  three  levels:  low  (one  UAV),  medium  (two  UAVs),  and  high 
(five  UAVs). 

Second,  Operational  Sequence  Analysis.  To  understand  the  logical  interconnection  of  operators’  tasks  and  the 
flow  of  information  throughout  the  system  during  the  conduct  of  the  mission  scenario,  operational  sequence 
diagrams  (OSDs)  were  used  to  show  the  flow  of  information  and  operator  functions  through  the  system  in 
relation  to  the  mission  timeline.  The  visual  representation  of  OSDs  indicated  actions,  inspections,  data 
manipulation  (i.e.,  transmission,  reception,  and  storage),  time  delays,  and  decisions  of  the  mission  scenario. 
Thus,  OSDs  are  particularly  useful  for  the  analysis  of  highly  complex  systems  that  require  many  time -critical 
information-decision-action  functions  by  multiple  users,  which  is  the  case  for  this  study  [22],  With  the 
scenario  written,  a  series  of  OSDs  were  prepared  to  facilitate  a  function  and  goal/task  analysis  of  the 
envisaged  system  and  the  development  of  a  model  from  which  performance  predictions  could  be  made  and  the 
potential  for  IAI  automation  agents  to  be  identified.  These  OSDs  also  created  an  inventory  of  all  bottom-up 
lowest  level  tasks  in  a  temporal  sequence.  A  small  portion  of  an  OSD  as  an  example  of  the  operational 
network  between  three  UAV  operators  is  shown  in  Figure  3. 
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Part  1  Scenario  of  UAV  Operation  Page  26 

Min  1-20  Control  of  single  VTUAV  during  search  for  terrorist  vessel 
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^located  on  VTUAV  radar 

Figure  3.  A  sample  of  Operational  Sequence  Diagrams  [22] 

Third,  hierarchical  goal  analysis  (HGA).  Research  on  IAIs  in  the  military  research  and  development 
community  was  limited  when  this  work  was  conducted.  As  a  result,  the  work  began  with  an  HGA  within 
which  standard  mission,  operation,  and  goal  analysis  procedures  were  followed  to  gain  more  detailed 
understanding  of  implementation  issues  and  opportunities  for  IAI  tasks  that  can  be  automated.  A  particular 
interest  in  this  work  is  the  identification  of  goals  that  are  candidates  for  IAI  automation  agents.  Using  the  AIO 
principles  and  IAI  framework,  the  agent,  user,  system,  and  world  models  were  created.  Upon  the  completion 
of  the  scenario  and  OSDs,  a  hierarchical  decomposition  of  the  envisaged  system  goals  was  conducted.  The 
goal  decomposition  for  all  three  UAV  operators  took  place  according  to  a  means-end  hierarchy  and  the  needs 
of  analysis  were  typically  satisfied  at  the  fourth  or  fifth  level  in  this  exercise.  The  goal  decomposition  was  first 
performed  in  a  top-down  fashion  from  the  highest  level  (e.g.,  GOAL=  counter-terrorist  mission  is  completed) 
down  to  lower  levels  (e.g.,  GOAL=  Vertical  Take-off  UAV  sector  search  is  planned).  Then,  a  more  stringent 
bottom-up  approach  was  completed  with  the  study  of  the  detailed  mission  activities  in  OSDs.  As  a  result, 
more  goals  were  added  in  the  top-down  analysis  list  to  complete  the  generation  of  the  HGA  inventory.  An 
example  of  a  small  portion  of  HGA  results  is  illustrated  in  Table  1 . 
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Table  1.  An  Example  of  HGA  Results  [22] 


Number 

Level 

Goal/Objective  and  Sub-goals/Sub-objectives 

IAI 

Candidate 

Influenced 

Variable 

Assignment 

Completion 
Time  (Sec) 

1 

2 

3 

4  5 

Top 

1  want  to  perceive  the  (...)  conduct  of  the  terrorist  patrol  mission 

9 

...  communications  are  conducted  and  maintained 

9.1 

...  directions  (instructions)  are  received 

9.1.1 

...  directions  are  received  from  other  crew  members 

Yes 

Directions 

UAV  Pilot 

4  to  25 

9.1.2 

...  directions  are  received  from  other  units 

Yes 

Directions 

UAV  Pilot 

5  to  14 

9.2 

information  is  received 

9.2.1 

...  information  is  received  from  other  crew  members 

9.2.1. 1 

...  VTUAV  refuelling  location 

Location 

Tactical 

Navigator 

5 

9.2. 1.2 

...  VTUAV  calculated  time  on  task 

Yes 

Time 

Tactical 

Navigator 

5 

9.2. 1.3 

...  the  flight  crew's  message  that  contact  is 
identified 

Message 

Sensor 

Operator 

5 

Fourth,  task  network  modelling  and  simulation.  Following  the  generation  of  OSDs  and  the  HGA,  tasks  related 
to  a  system  that  can  be  automatically  operated  (with  automation  intelligence  or  expert  knowledge)  were 
chosen  to  be  appropriate  IAI  automation  agents.  Goals  for  systems  that  have  no  information  on  automated 
operations  could  not  be  identified  as  IAI  automation  tasks.  With  the  identification  of  these  IAI  tasks/agents,  a 
task  network  model  was  developed  in  an  Integrated  Performance  Modelling  Environment  (IPME).  IPME  is  a 
discrete  event-simulation  framework  used  to  model  and  assess  human  and  system  performance  [8].  It 
comprised  an  environment  model,  a  crew  model,  a  task  network  model,  a  performance-shaping  model,  and 
optional  external  models.  Combined  with  IPME’s  scheduling  algorithm,  these  models  can  help  an  analyst 
predict  workload  and  operator  performance.  An  IPME  database  was  also  generated  while  producing  the  OSD 
and  HGA  tasks.  In  the  database,  each  task  was  allocated  to  an  operator  or  a  system  with  a  descriptive  label. 
With  the  database,  a  sequential  type  of  operational  network  model  was  developed.  Although  the  network 
model  was  a  UAV  operator  model,  external  events  (other  aircrew  activities,  UAV  activities,  and  other  unit 
activities)  had  to  be  established  to  allow  the  network  to  function  as  a  closed-loop  feedback  system.  These 
were  prepared  and  included  in  the  network  which  was  used  to  define  task  behaviour,  operator  assignment,  and 
interactions  between  tasks  and  operators.  By  linking  together  various  networks  or  tasks,  the  model  attempts  to 
replicate  the  behaviour  of  a  real  system.  The  task  network  model  was  run  10  times  in  IPME  to  simulate  the 
mission  conducted.  The  simulation  data  collected  were  used  to  analyze  operators’  behaviours  and  determine 
the  effects  of  IAI  agents  on  operators’  performances.  The  IAI  models  provided  not  only  an  effective  means  to 
assess  the  merits  of  incorporating  new  automation  technologies,  but  also  a  clear  indication  of  the  most  fruitful 
areas  for  further  research  and  developments. 

Fifth,  the  human-in-the-loop  experimentation.  The  IAI  framework  and  related  AIO  principles  were  used  to 
analyze  mission  scenarios,  identify  IAI  tasks,  and  generate  task  network  models.  The  simulation  study  was 
considered  as  initial  estimates  of  IAI  agents  being  augmented  decision-aids,  but  further  analysis  regarding  the 
most  beneficial  IAI  tasks  to  modify  was  required.  Additionally,  considerable  effort  was  still  needed  to  apply 
IAI  framework  and  related  AIO  principles  to  design  prototype  systems.  Strong  empirical  evidence  was  also 
required  to  substantiate  this  effort.  Thus,  an  experimental  synthetic  environment  (SE)  was  designed  and 
developed  to  conduct  a  human-in-the-loop  experiment  and  validate  the  task  network  modelling  method  used 
in  the  simulation.  Consistent  with  the  UAV  crew  positions  used  in  the  performance  modelling  phase,  the  SE 
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had  three  control  consoles  replicating  CP140  tactical  compartment  multifunction  workstations.  The 
workstations  were  designed  to  communicate  with  virtual  UAVs  through  fully  functional,  real  world  software 
interfaces.  Each  of  them  had  a  set  of  appropriate  displays  and  controls  for  the  UP,  UO,  and  TN  as  illustrated  in 
Figure  4.  The  experimental  environment  also  had  the  ability  to  integrated  video  and  audio  data  collection,  thus 
enabling  empirical  assessments  of  IAI  concepts  developed  in  the  first  phase. 


Figure  4:  Experimental  Environment  Showing  Three  UAV  Control  Stations  for  UAV  Pilot,  Sensor 

Operator,  and  Tactical  Navigator  [26]. 


The  IAI  agents  were  functional  components  of  the  UAV  control  SE  developed  for  this  research.  They 
supported  the  participants  in  accomplishing  the  assigned  mission  tasks  of  the  experiment  by  providing 
decision  support  to  the  crew  and  by  taking  over  certain  high  workload  crew  tasks.  A  3  x  3  x  2  (Operator 
Workload:  one  UAV  vs.  two  UAVs  vs.  three  UAVs;  Operator  Position  Complexity:  UP  vs.  UO  vs.  TN;  IAI 
Interface  Condition:  ON  vs.  OFF)  mixed-factor  design  was  used  in  the  experiment,  as  illustrated  in  Figure  5. 
The  IAI  for  each  crewmember  was  tailored  to  suit  their  individual  needs.  Both  Operator  Workload  and  IAI 
Interface  Condition  were  within-subjects  factors,  meaning  that  participants  were  tested  under  all  levels  of 
those  factors,  whereas  Position  Complexity  was  a  between-subjects  factor  (i.e.,  crewmembers  remained  at  one 
position  throughout  the  experiment). 
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Figure  5:  Depiction  of  the  Mixed-Factor  Experimental  Design  [26]. 


4.5  Discussion  and  Implications 

Although  the  IAI  implemented  as  a  prototype  system  in  this  case  study  was  only  a  small  subset  of  a  future, 
more  extensive  suite  of  a  fully  optimized  agent  system,  the  experimental  findings  indicated  that  the  control  of 
a  dynamic  and  complex  system  such  as  multiple  disparate  UAV  control  from  an  airborne  platform  can  be 
improved  through  the  use  of  a  multi-agent  IAI  suite.  More  importantly,  through  the  discussions  and 
observations  made  during  the  conduct  of  the  study,  experience  and  knowledge  were  gained  regarding  the  use 
of  IAI  framework  and  associated  AIO  principles  for  the  design  of  IAI  agents,  the  implementation  of  synthetic 
IAI  prototype  environments,  and  the  conduct  of  the  experiments.  Many  thoughts  were  also  given  to  the  details 
of  IAI  design  issues  which  were  discussed  in  Hou,  Kobierski,  and  Herdman  [26]. 

This  case  study  on  IAI  design  for  the  control  of  multiple  UAVs  demonstrated  that  the  proposed  IAI 
framework  and  associated  AIO  principles  provided  useful  guidance  for  IAI  developments.  The  framework  is 
complete',  the  principles,  the  models  and  the  analytical  methodologies  covered  all  the  tasks  related  to  the  entire 
design  and  evaluation  cycle  of  a  human-machine  system;  consistent',  the  principles,  the  models  and  the 
analytical  methodologies  keep  a  consistent  track  from  abstract  guidelines  to  concrete  models  and  methods; 
effective:  the  framework  speeds  up  the  design  and  execution  of  the  experiment  in  both  time  and  personal 
management  because  of  the  clear  division  between  the  principles  and  models;  and  user-friendly :  the  analytical 
methodologies  in  Figure  2  provides  clear  processes  for  the  system  design. 

With  the  successful  use  of  IAI  framework  in  the  operator  interface  design  for  the  control  of  multiple  UAVs, 
related  design  principles  are  being  used  in  broader  fields  including  operator  interface  design  for  the  control  of 
an  extremely  agile  micro-aerial  vehicle  (EA-MAV).  Having  an  EA-MAV  that  can  be  controlled  in-theatre 
means  an  infantry  section  can  receive  reconnaissance  quickly  and  have  information  delivered  to  operators 
who  need  it  most.  To  best  use  such  technology,  operators  require  a  ground  control  system  (GCS)  to  interface 
with  the  flying  machine  they  are  controlling.  The  GCS  itself  is  subject  to  stringent  engineering  criteria,  such 
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as  the  need  to  be  robust  enough  to  withstand  heat  extremities,  moisture,  and  impact.  Additionally,  an  OMI 
must  be  intuitive  in  function  and  easy  to  learn  for  average  soldiers.  If  the  OMI  is  not  optimized,  there  is 
potential  for  loss  of  systems  and  time  during  mission-critical  moments  as  operators  attempt  to  decipher  what 
the  next  control  sequence  should  be.  Further,  the  GCS  must  be  small  and  light  enough  to  avoid  serious  burden 
to  the  operator  who  will  be  carrying  it.  Keeping  these  requirements  in  mind,  LAI  framework  is  being  used  to 
design  and  develop  an  operator  interface  on  handheld  devices.  Similar  to  the  design  process  of  IAI  for  the 
control  of  multiple  UAVs,  a  mission  scenario  was  developed  to  represent  the  typical  use  of  EA-MAV  for 
infantry  solders.  This  time,  cognitive  task  analysis  was  used  to  identify  function  groups  for  IAI  automation 
agents.  AIO  principles  were  used  to  develop  user,  task,  system,  environment,  and  agent  models.  Figure  6 
shows  a  snapshot  of  an  OMI  on  a  tablet  PC. 


EA-MAV  CONTROL  SYSTEM 

Mission  Loaded  Mission _ 


MAV 

e*  a 


r ■■sai  I 

6*r&y  I 


Map  Options 

Mission  Tasks 

Overlays: 

Mission  Path: 

/Gnd  Lines 

/  MAV  Path 

X  1662,  Y  1300 

■/Video  Field 

X  1646  Y  900 

X  1545,  Y  611 

X  1573,  Y  591 

X  1538,  Y  360 

X  1343,  Y  433 

X  1180,  Y  568 

X  1102,  Y  455 

X  1202,  Y  361 

X  1308,  Y  441 

X  1400,  Y  560 

Figure  6.  A  Snapshot  of  OMI  on  a  Tablet  PC  for  the  Control  of  EA-MAV  [21]. 

The  prototype  interfaces  will  connect  to  a  synthetic  3D  environment  where  a  virtual  EA-MAV  can  be 
controlled  by  handheld  devices.  An  example  plan  for  an  empirical  investigation  was  developed  to  determine 
how  using  a  device  with  tactile  hardware  buttons,  as  opposed  to  a  touch  screen  input  device,  affects  the 
performance  of  an  EA-MAV  operator.  The  findings  will  not  only  assist  in  choosing  the  optimal  input  and 
display  method  and  thus  an  effective  OMI  for  the  final  GCS  system,  but  will  also  provide  design  guidelines 
for  the  control  of  EA-MAVs  using  handheld  devices.  The  design  processes  will  also  validate  the  IAI 
framework  and  associated  principles  for  OMI  design. 
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5.0  CONCLUSIONS  AND  FUTURE  WORK 

The  IAI  framework  provides  guidance  to  the  effective  design  of  OMI.  Through  the  IAI  design  case  study  for 
the  control  of  multiple  UAVs,  IAI  framework  and  associated  AIO  principles  have  proven  to  be  an  effective 
design  tool  to  guide  the  design  process  and  thus  enhance  human-machine  system  performance.  They 
contributed  to  the  Canadian  Forces  Air  Force  automation  policy  and  planning  developments.  They  are  being 
used  to  design  OMIs  for  the  control  of  an  EA-MAV  to  address  situation  awareness  in  an  urban  operation 
scenario.  Although  many  analytical  approaches  and  design  models  have  been  used,  there  are  still  some  models 
that  have  not  yet  been  completed  (e.g.,  human  cognition  model  for  physiological  monitoring)  due  to  the 
immaturity  of  relevant  technologies  (e.g.,  Augmented  Cognition).  Thus,  IAI  framework  and  associated  AIO 
principles  still  need  to  be  further  developed  and  validated  within  different  contexts.  More  empirical  studies  are 
still  required  to  enhance  the  robustness  of  this  design  tool.  To  accomplish  this  goal,  IA  framework  will  be 
applied  to  design  OMIs  for  a  complex,  synthetic  environment  where  a  network-enabled  operations  approach 
will  be  used  for  the  control  of  various  UVs.  The  IAI  framework  will  also  guide  a  study  of  multi-modal  OMI 
design  for  the  control  of  UV  swarming  to  reduce  human  error  and  improve  human  performance.  These 
investigations  will  further  validate  IAI  design  principles  and  refine  customized  approaches  for  different 
applications.  The  revised  principles  will  contribute  to  the  standardization  of  operator  interface  design 
principles  and  facilitate  interoperability  across  many  military  systems. 


ACKNOWLEDGMENT 

The  authors  thank  Sheila  Young  for  the  editing. 


REFERENCES 

[1]  Abascal,  J.,  I.  F.  d.  Castro,  et  al.  (2008).  Adaptive  Interfaces  for  Supportive  Ambient  Intelligence 
Environments  Computers  Helping  People  with  Special  Needs.  Berlin,  Germany,  Springer.  5105:  30- 
37. 

[2]  Banbury,  S.,  M.  Gauthier,  et  al.  (2007).  Intelligent  Adaptive  Systems:  Literature  Research  of  Design 
Guidance  for  Intelligent  Adaptive  Automation  and  Interfaces.  Defence  Research  &  Development 
Canada  -  Toronto. 

[3]  Benyon,  D.  R.  &  D.  M.  Murray  (1988).  Experience  with  Adaptive  Interfaces.  The  Computer  Journal 
31(5):  465-473. 

[4]  Benyon,  D.  R.  &  D.  M.  Murray  (1993).  Adaptive  Systems:  From  Intelligent  Tutoring  to  Autonomous 
Agents.  Knowledge-Based  Systems  6(4):  197-219. 

[5]  Benyon,  D.  R.  &  Murray,  D.  M.  (2000).  Special  Issue  on  Intelligent  Interface  Technology:  Editor's 
Introduction.  Interacting  with  Computers  12(4):  315-322. 

[6]  Bickmore,  T.  W.,  &  Picard,  R.  W.  (2005).  Establishing  and  Maintaining  Long-Term  Human- 
Computer  Relationships.  ACM  Transactions  on  Computer-Human  Interaction,  12(2),  293-327. 

[7]  Browne,  D.,  P.  Totterdell,  et  al.  (1990).  Adaptive  User  Interfaces.  London,  UK,  Academic  Press. 


31  -16 


RTO-MP-HFM-1 81 


Intelligent  Adaptive  Interface:  A  Design  Tool 
for  Enhancing  Human-Machine  System  Performances 


[8]  D.  Dahn  &  K.  R.  Laughery  (1997).  The  Integrated  Performance  Modelling  Environment  —  Simulating 
Human-System  Performance.  In  S.  Andradottir,  K.  J.  Healy,  D.  H.  Withers,  and  B.  L.  Nelson  (Eds.), 
Proceeding  of  the  1997  Winter  Simulation  Conference  (pp.  1141-1145).  December  1997,  Atlanta, 
GA. 

[9]  Edwards,  J.  L.  (1990).  Proposed  Architecture  for  SDBMS-2.  Toronto,  Ontario,  Canada:  Report 
prepared  for  the  Defence  and  Civil  Institute  of  Environmental  Medicine  (DCIEM)  by  Artificial 
Intelligence  Management  and  Development  Corporation. 

[10]  Edwards,  J.  L.  (2000).  Plans  and  Goals  in  A  Testbed  for  Intelligent  Aiding  -  Final  Report.  (Tech. 
Rep.  W771 1-8-7564).  Toronto,  Ontario:  Defence  and  Civil  Institute  of  Environmental  Medicine 
(DCIEM). 

[11]  Edwards,  J.  L.  (2004).  A  Generic,  Agent-Based  Framework  for  the  Design  and  Development  of 
UAV/UCAV  Control  Systems.  Defence  Research  &  Development  Canada  -  Toronto. 

[12]  Emerson,  T.  J.,  M.  Rienecke,  et  al.  (1988).  The  Human  -  Electronic  Crew:  Can  They  Work  Together? 
Farnborough  Airfield,  Hampshire,  UK,  RAF  Institute  of  Aviation  Medicine. 

[13]  Emerson,  T.  J.,  M.  Rienecke,  et  al.  (1990).  The  Human  -  Electronic  Crew:  Is  The  Team  Maturing? 
Farnborough  Airfield,  Hampshire,  UK,  RAF  Institute  of  Aviation  Medicine. 

[14]  Findlater,  L.  and  J.  McGrenere  (2004).  A  Comparison  of  Static,  Adaptive,  and  Adaptable  Menus. 
Proc.  of  the  ACM  Conference  on  Human  Factors  in  Computing  Systems.  Vienna,  Austria:  89-96. 

[15]  Franklin,  D.,  J.  Budzik,  et  al.  (2002).  Plan-based  Interfaces:  Keeping  Track  of  User  Tasks  and  Acting 
to  Cooperate.  International  Conference  on  Intelligent  User  Interfaces:  Avail: 
http://www.iuiconf.org/02pdf/2002-00 1  -00 1 6.pdf. 

[16]  Hancock,  P.  A.  &  M.  H.  Chignell  (1988).  Mental  Workload  Dynamics  in  Adaptive  Interface  Design. 
IEEE  Transactions  on  Systems,  Man,  and  Cybernetics  18(4):  647-658. 

[17]  Hayes-Roth,  B.  (1991).  Making  Intelligent  Systems  Adaptive.  Architectures  for  Intelligence:  The 
Twenty-Second  Carnegie  Mellon  Symposium  on  Cognition.  K.  VanLehn.  Hillside,  NJ,  Lawrence 
Erlbaum  Associates:  301-321. 

[18]  Haylock,  T.  M.  (2008).  A  Prototype  User  Interface  for  the  Control  of  Extremely  Agile  Micro-Aerial 
Vehicles.  UVS  Canada  2008  Conference,  Ottawa.,  Canada. 

[19]  Hilburn,  B.,  Molloy,  R.,  Wong,  D.,  and  Parasuraman,  R.  (1993).  Operator  Versus  Computer  Control 
of  Adaptive  Automation.  Technical  Report  NAWACADWAR-93031-60  Naval  Air  Warfare  Center, 
Warminster,  PA,  1993. 

[20]  Hook,  K.  (2000).  Steps  to  Take  Before  Intelligent  User  Interfaces  Become  Real.  Interacting  with 
Computers  12(4):  409-426. 


RTO-MP-HFM-181 


31  - 17 


Intelligent  Adaptive  Interface:  A  Design  Tool 

for  Enhancing  Human-Machine  System  Performances 


[21]  Hou,  M.,  Keillor,  J.,  Wong,  F.,  Haylock,  T.  M.,  &  Somjee,  K.  (2009).  Development  of  Prototype 
Interfaces  for  the  Control  of  a  Micro-Aerial  Vehicle.  International  Ergonomics  Association  2009 
Congress,  Beijing,  China. 

[22]  Hou,  M.  &  Kobierski,  R.  D.  (2005).  Performance  Modeling  of  Agent-Aided  Operator-Interface 
Interaction  for  the  Control  of  Multiple  UAVs.  Proceedings  of  the  2005  IEEE  International  Conference 
on  Systems,  Man,  and  Cybernetics  (pp.  2463-2468),  Waikoloa,  Hawaii,  USA. 

[23]  Hou,  M.  &  Kobierski,  R.  D.  (2006a).  Experimental  Assessment  of  Intelligent  Adaptive  Interfaces  for 
UAV  Operators.  (Contract  Rep.  CR  2006-207).  Defence  Research  &  Development  Canada  -  Toronto. 

[24]  Hou,  M.  &  Kobierski,  R.  D.  (2006b).  Operational  Analysis  and  Performance  Modeling  for  the 
Con  trol  of  Multiple  Uninhabited  Aerial  Vehicles  from  an  Airborne  Platform.  In  Salas,  E  (Series  Ed.) 
&  N.  J.  Cooke,  H.  L.  Pringle,  H.  K.  Pedersen,  &  O.  Connor  (Vol.  Eds.)  Advances  in  Human 
Performance  and  Cognitive  Engineering  Research:  Vol.  7.  Human  Factors  of  Remotely  Operated 
Vehicles  (pp.  267-282).  Amsterdam,  The  Netherlands:  Elsevier. 

[25]  Hou,  M.,  Kobierski,  R.  D.  et  al.  (2007).  Intelligent  Adaptive  Interfaces  for  the  Control  of  Multiple 
UAVs.  Journal  of  Cognitive  Engineering  and  Decision  Making  1(3):  327-362. 

[26]  Hou,  M.,  Kobierski,  R.,  &  Herdman,  C.  M.  (2006).  Design  and  Evaluation  of  Intelligent  Adaptive 
Operator  Interfaces  for  the  Control  of  Multiple  UAVs.  NATO  RTO  HFM  135  Symposium  2006, 
Biarizze,  France. 

[27]  Innocent,  P.  R.  (1982).  Towards  Self-Adaptive  Interface  Systems.  International  Journal  of  Man- 
Machine  Studies  16(3):  287-299. 

[28]  Jaquero,  V.  L.,  F.  Montero,  et  al.  (2009).  Intelligent  User  Interfaces:  Past,  Present  and  Future. 
Engineering  the  User  Interface:  From  Research  to  Practice.  C.  B.  Miguel  Redondo,  Manuel  Ortega. 
London,  Springer:  259-270. 

[29]  Kauffman,  S.  A.  (1994).  Whispers  From  Carnot:  The  Origins  of  Order  and  Principles  of  Adaptation 
in  Complex  Nonequilibrium  Systems.  Complexity:  Metaphores,  Models,  and  Reality.  G.  Cowan,  D. 
Pines  and  D.  Meltzer.  Reading,  MA,  Addison-Wesley.  XIX:  83-160. 

[30]  Khoshnevis,  B.  &  Austin,  W.  (1987).  An  Intelligent  Interface  for  System  Dynamics  Modeling.  The 
Conference  on  Al  and  Simulation.  San  Diego,  CA:  81-86. 

[31]  Kolenda,  T.  (2002).  Adaptive  Tools  in  Virtual  Environments  Independent  Component  Analysis  for 
Multimedia.  Unpublished  PhD,  Technical  University  of  Denmark,  Lyngby,  Denmark. 

[32]  Lamberti,  D.  M.  &  Wallace,  W.  A.  (1990).  Intelligent  interface  design:  An  empirical  assessment  of 
knowledge  presentation  in  expert  systems.  MIS  Quarterly  14(3):  279-311. 

[33]  Langley,  P.  (1999).  User  Modelling  in  Adaptive  Interfaces.  The  7th  International  Conference  on  User 
Modelling.  Banff,  Alberta:  357-370. 


31  -18 


RTO-MP-HFM-1 81 


Intelligent  Adaptive  Interface:  A  Design  Tool 
for  Enhancing  Human-Machine  System  Performances 


[34]  Lehner,  P.  E.,  Cohen,  M.  S.,  et  al.  (1987).  Adaptive  Decision  Aiding:  Interim  Report.  Falls  Church, 
VA,  Decision  Science  Consortium. 

[35]  Lipa,  J.  F.  (1999).  Application  of  Advanced  Displays  and  Intelligent  Interfaces  Technology.  Rome, 
NY,  AFRF/IFSA. 

[36]  Fiu,  J.,  Wong,  C.  K.,  et  al.  (2003).  An  Adaptive  User  Interface  Based  on  Personalized  Learning.  IEEE 
Intelligent  Systems  18(2):  52-57. 

[37]  Maat,  L.  &  Pantic,  M.  (2007).  Gaze-X:  Adaptive,  Affective,  Multimodal  Interface  for  Single-User 
Office  Scenarios.  Artificial  Intelligence  for  Human  Computing,  LNCS.  Berlin,  Germany,  Springer. 
4451:251-271. 

[38]  McTear,  M.  F.  (1993).  User  Modelling  for  Adaptive  Computer  Systems:  A  Survey  of  Recent 
Developments.  Artificial  Intelligence  Review  7(0):  157-184. 

[39]  McTear,  M.  F.  (2000).  Intelligent  Interface  Technology:  From  Theory  to  Reality?  Interacting  with 
Computers  12(0):  323-336. 

[40]  Meyer,  B.,  Yakemovic,  K.  C.  B.,  et  al.  (1993).  Issues  in  Practical  Application  of  an  Adaptive 
Interface.  The  First  International  Conference  on  Intelligent  User  Interfaces.  Orlando,  Florida,  USA: 
251-254. 

[41]  Miller,  C.  A.,  Pelican,  M.,  &  Goldman,  R.  (1999).  Tasking  Interfaces  for  Flexible  Interaction  with 
Automation:  Keeping  the  Operator  in  Control.  In  the  Proc.  of  the  International  Conference  on 
Intelligent  User  Interfaces,  Redondo  Beach,  CA,  USA. 

[42]  Mitrovic,  N.,  Royo,  J.  A.,  et  al.  (2007).  Performance  Analysis  of  an  Adaptive  User  Interface  System 
Based  on  Mobile  Agents.  Engineering  Interactive  Systems.  J.  Gulliksen,  M.  B.  Haming,  P.  Palanque, 
G.  C.  v.  d.  Veer  and  J.  Wesson.  Berlin,  Germany,  Springer:  1-18. 

[43]  Morris,  N.  M.  &  Rouse,  W.  B.  (1986).  Adaptive  Aiding  for  Human  Computer  Control:  Experimental 
Studies  of  Dynamic  Task  Allocation.  (AFAMRL-TR-86-0005). 

[44]  National  Institute  of  Standards  and  Technology.  (1993).  Integrated  Definition  for  Function  Modeling 
(IDEF0).  Springfield,  VA. 

[45]  Noah,  W.  W.  &  Halpin,  S.  M.  (1986).  Adaptive  User  Interfaces  for  Planning  and  Decision  Aids  in 
C3 1  Systems.  IEEE  Transactions  on  Systems,  Man  and  Cybernetics  16(6):  909-918. 

[46]  Norcio,  A.  F.  &  Stanley,  J.  (1989).  Adaptive  Human-Computer  Interfaces:  A  Literature  Survey  and 
Perspective.  IEEE  Transactions  on  Systems,  Man  and  Cybernetics  19(2):  399  -  408. 

[47]  Parasuraman,  R.,  Mouloua,  M.,  et  al.  (1996).  Effects  of  Adaptive  Task  Allocation  on  Monitoring  of 
Automated  Systems.  Human  Factors  38(4):  665-679. 

[48]  Pew,  R.  W.  &  Mavor,  A.  S.  (2007).  Human-System  Integration  in  the  System  Development  Process: 
A  New  Loo.  Washington,  D.C.,  The  National  Academies  Press. 


RTO-MP-HFM-181 


31  - 19 


Intelligent  Adaptive  Interface:  A  Design  Tool 

for  Enhancing  Human-Machine  System  Performances 


mUtiAiUtat  Artt/rt  kt 


[49]  Powers,  W.  T.  (1990).  A  Hierarchy  of  Control.  In,  R.  J.  Robertson  and  W.  T.  Powers  (Eds.), 
Introduction  to  Modern  Psychology:  The  Control-Theory  View.  Gravel  Switch,  KY:  The  Control 
Systems  Group  Inc.,  59-82. 

[50]  Rouse,  W.  B.,  Geddes,  N.  D.,  et  al.  (1987).  An  Architecture  for  Intelligent  Interfaces:  Outline  of  an 
Approach  to  Supporting  Operators  of  Complex  Systems.  Human-Computer  Interaction  3(2):  87-122. 

[51]  Schereiber,  G.,  Akkermans,  H.,  Anjewierden,  A.,  de  Hoog,  R.,  Shadbolt,  N.,  Van  de  Velde,  W.,  & 
Wielinga,  W.  (2000).  Knowledge  Engineering  and  Management:  The  CommonKADS  Methodology. 
Paper  presented  at  the  MIT  Press,  Cambridge,  Massachusetts. 

[52]  Seaward,  M.  R.  (1998).  Interactive  Assistants  Provide  Ease  of  Use  for  Novices:  The  Development  of 
Prototypes  and  Descendants.  Computers  in  Human  Behavior,  14(2),  221-237. 

[53]  Sukaviriya,  P.  (1993).  From  User  Interface  Design  to  the  Support  of  Intelligent  and  Adaptive 
Interfaces:  An  Overhaul  of  User  Interface  Software  Infrastructure.  Knowledge-Based  Systems  6(4): 
220-229. 

[54]  Taylor,  R,  M.  (1997).  Human  Electronic  Crew  Team  Work.  In  M.  J.  Smith,  G,  Salvendy,  and  R.  J. 
Koubek  (Eds).  Design  of  Human-Computer  Systems:  Social  and  Ergonomic  Considerations.  Elsevier: 

[55]  Tenney,  Y.  J.,  Rogers,  W.  H.,  et  al.  (1995).  Pilot  Opinions  on  High  Level  Flight  Deck  Automation 
Issues:  Towards  the  Development  of  a  Design  Philosophy.  Cambridge,  MA.  NASA. 

[56]  Tomlinson,  G.-B.,  Baumer,  E.,  et  al.  (2007).  Dreaming  of  Adaptive  Interface  Agents.  ACM 
Conference  on  Human  Factors  in  Computing  Systems.  San  Jose,  CA,  USA  ACM:  2007  - 
20 12. Amsterdam,  21B,  247-250. 

[57]  Tyler,  S.  W.  (1990).  Adaptive  Interfaces.  Palo  Alto,  CA,  Lockheed  Al  Center. 

[58]  Vicente,  K.  J.,  &  Rasmussen,  J.  (1992).  Ecological  Interface  Design:  Theoretical  Foundations.  IEEE 
Transactions  on  Systems,  Man,  and  Cybernetics,  22  (4),  589-606. 

[59]  Yoon,  W.  C.  &  Y.  S.  Kim  (1996).  Aiding  the  Analysis  of  Human  Actions  in  Large-Scale  Systems:  An 
Intelligent  Interface  Approach.  Computers  and  Engineering  30(3):  569-577. 


31  -20 


RTO-MP-HFM-1 81 


